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FOREWORD 

This  research  and  development  was  performed  under  Work  Unit  Z1176-PN.01 
(Improving  the  Navy's  Computer-Managed  Training  System),  as  part  of  a  project  to 
improve  the  Navy's  operational  computer-managed  instruction  (CMI)  system.  It  was 
sponsored  by  the  Deputy  Chief  of  Naval  Operations  (OP-01). 

This  is  the  third  in  a  series  of  reports  on  Navy  CMI.  The  first,  NPRDC  SR  80-33, 
described  the  CMI  system  and  the  development  of  the  R&D  project  itself.  The  second, 
NPRDC  TR  81-6,  described  the  effects  of  alternative  student- to-instructor  ratios  in  CMI 
learning  centers  on  student  performance  and  instructor  behavior.  The  purpose  of  the  work 
described  herein  was  to  determine  how  the  automated  scoring  of  teletypewriting  tests 
would  affect  training.  Results  will  be  used  by  the  Chief  of  Naval  Education  and  Training 
(CNET),  the  Chief  of  Naval  Technical  Training  (CNTT),  commanding  officers  of  all  the 
Navy  CMI  schools,  and  others  concerned  with  computer-based  instruction. 

Appreciation  is  expressed  to  personnel  at  the  Radioman  "A"  School  at  the  Naval 
Training  Center,  San  Diego,  and  to  the  staff  at  the  Instructional  Program  Development 
Center,  San  Diego,  for  developing  the  automated  teletyping  testing;  to  the  Management 
Information  and  Instructional  Systems  Activity,  Memphis,  for  its  computer  support;  and  to 
the  CMI  System  Manager,  CNTT,  for  his  support  in  managing  the  many  operational 
elements  of  this  R&D  task. 


3AMES  F.  KELLY,  JR. 
Commanding  Officer 


JAMES  3.  REGAN 
Technical  Director 


SUMMARY 


Problem  and  Background 

A  goal  of  the  Navy's  computer-managed  instruction  (CMI)  system  is  to  train  students 
to  meet  specific  performance  objectives  in  the  shortest  possible  time.  Although  testing 
of  a  student's  knowledge  has  been  automated,  most  performance  skills  are  still  manually 
tested  and  scored.  With  these  manual  procedures,  testing  sessions  are  lengthy,  scoring  is 
inaccurate,  and  no  records  of  errors  are  kept.  This  lack  of  detailed  information  on  past 
performance  error  hampers  remediation  efforts,  and  prolongs  training  time. 

A  system  for  automated  performance  testing  (APT)  has  been  implemented  in  the 
teletypewriter  portion  of  the  Radioman  "A”  School,  San  Diego.  This  system  includes  a 
computer-generated  Error  Distribution  Report  (EDR)  that  provides  detailed  feedback  on 
student  typing  errors. 

Objectives 

The  objectives  of  this  effort  were: 

1.  To  determine  whether  test-related  activities  take  less  time  under  APT  than 
under  manual  testing  conditions. 

2.  To  determine  whether  training  time  can  be  reduced  by  different  applications  of 
EDR  use,  and,  if  so,  the  most  feasible  and  effective  techniques  for  distributing  EDRs  to 
students  and  instructors. 

Approach 

Two  experiments  were  conducted  to  investigate  the  influence  of  APT  and  EDR  on  the 
TTY  portion  of  the  Radioman  "A"  School  training  program.  In  the  first  experiment,  the 
times  required  to  perform  test-related  activities  under  manual  scoring  were  compared 
with  those  required  to  perform  such  activities  under  automated  conditions.  In  the  recond 
experiment,  students  in  one  group  received  EDRs  daily;  and  those  in  another  group,  only 
during  their  second  remedial  session.  Those  in  the  third  group  received  no  reports.  The 
numbers  of  hours,  tests,  and  sessions  required  to  reach  specific  performance  levels  were 
measured.  Also,  student  attitudes  toward  CMI,  APT,  and  EDRs  were  obtained  through  a 
random  survey. 

Results 

1.  Testing  under  the  APT  procedures  was  faster  than  manual  testing  and  grading. 

2.  Students  in  the  group  receiving  EDRs  daily  finished  the  typing  portion  of  the 
course  nearly  3  days  faster  than  did  students  who  did  not  receive  the  reports. 

3.  The  attrition  rate  for  the  group  receiving  EDRs  daily  was  15  percent,  compared 
to  35  percent  for  the  group  receiving  no  EDRs. 

4.  A  majority  of  students  favored  the  CMI  system  in  general,  and  the  APT  program 
in  particular.  Students  unanimously  approved  the  use  of  EDRs. 
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Conclusions 


It  appears  that  APT  is  effective  in  reducing  absolute  test-taking  time,  and  that  EDRs 
could  reduce  both  overall  training  time  and  attrition. 

Recommendations 


1.  The  RM  school  should  continue  to  use  APT  and  the  EDRs  on  a  daily  basis. 

2.  The  Chief  of  Naval  Technical  Training  and  the  Service  School  Commands  should 
apply  APT  procedures  to  similar  typing  tests  at  other  Navy  technical  training  schools  with 
reasonable  assurance  of  improved  training  cost  effectiveness. 
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INTRODUCTION 


Problem 


The  Navy  has  attempted  to  meet  the  problem  of  rising  technical  training  costs 
through  the  development  of  the  computer-managed  (CMI)  system.  This  system  manages 
self-paced,  individualized  instruction  conducted  at  technical  training  schools  across  the 
continental  United  States  through  a  centralized  computer  located  at  the  Management 
Information  and  Instructional  Systems  Activity  (MIISA),  Millington,  TN.  Although  this 
innovation  in  large-scale  training  has  resulted  in  considerable  reductions  in  training  time, 
further  modifications  are  desirable  to  achieve  more  cost-effective  training.  Therefore, 
the  Navy  Personnnel  Research  and  Development  Center  (NAVPERSRANDCEN),  San 
Diego,  has  established  a  comprehensive  research  and  development  (R&D)  effort  directed 
at  making  the  CM1  system  an  even  more  efficient  component  of  Navy  technical  training. 
The  initial  report  on  this  research  effort  (Van  Matre,  1980)  described  the  problem  areas 
that  limit  the  effectiveness  of  the  CMI  system  and  plans  for  R&D  projects  that  will 
address  these  problem  areas.  The  second  report  (Van  Matre,  Hamovitch,  Lockhart,  & 
Squire,  1981)  described  the  effects  of  alternate  student  to  instructor  ratios  on  student 
performance  and  instructor  behavior. 

One  R&D  project  that  was  described  in  the  initial  report  concerned  the  development 
of  automated  performance  testing  (APT)  and  its  implementation  into  the  CMI  system. 
The  initial  effort  under  this  project  involves  the  automation  of  teletypewriter  (TTY) 
performance  testing  at  the  Radioman  "A"  school,  San  Diego.  The  instruction  course 
conducted  at  the  RM  school  was  developed  by  the  Instructional  Program  Development 
Center  (IPDC).  The  computer  program  that  automatically  times  and  scores  the  TTY  tests 
was  developed  by  MIISA.  NAVPERSRANDCEN  developed  the  diagnostic  typing  report 
described  by  this  research. 

Background 

The  RM  CMI  Course 


The  course  of  instruction  for  the  RM  school  consists  of  approximately  5  weeks  of 
individualized  instruction  in  common-core  knowledge  and  TTY  training,  followed  by  a 
practical  deck-watch  phase,  oriented  toward  either  sea  or  shore  duty,  lasting  2  to  3 
weeks.  The  common-core  phase  includes  training  in  the  use  of  basic  Navy  message  forms 
and  message  handling  procedures,  and  is  conducted  in  conjunction  with  daily  TTY  training 
periods.  In  the  deck-watch  phase,  the  student  demonstrates  typing,  message-handling,  and 
equipment-operating  skills  in  simulated  on-the-job  (sea  or  shore)  situations,  and  is 
required  to  successfully  complete  a  series  of  message-handling  and  equipment-operating 
routines. 

Actual  completion  times  for  the  course  vary  with  individual  students,  since  training  is 
self-paced,  and  aptitudes  and  skills  differ.  However,  management  of  the  two  phases  of 
training- -common-core  knowledge  and  TTY  training — is  structured  so  that  students  complete 
the  two  phases  at  about  the  same  time. 

TTY  Training  and  Testing  in  the  CMI  Course 

To  graduate  from  RM  school,  students  must  learn  teletyping  up  to  a  minimum 
performance  level  of  700  functions  (keystrokes)  in  a  5-minute  period,  with  no  more  than 
five  keystroke  errors.  This  criterion  level  is  equivalent  to  28  words  per  minute  of 
conventional  typing.  The  typical  student  has  had  no  previous  typing  instruction. 
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Basic  typing  instruction  takes  place  in  a  small  room  for  students  at  TTY  terminals 
supported  by  videotape  consoles.  The  students  are  given  up  to  eight  2-hour  sessions  to 
learn  the  keyboard  and  build  their  typing  speed  up  to  300  functions  per  5-minute  period 
(about  12  words  per  minute).  Any  time  students  feel  that  they  are  ready  to  qualify  at  a 
particular  level  of  performance,  they  may  request  a  test  assignment  from  the  computer. 
However,  during  the  ninth  session,  if  they  have  not  already  done  so,  students  must 
attempt  to  qualify  at  the  300  function  level.  An  additional  requirement  at  this  point  is 
that  certain  critical  lines  must  be  typed  with  100  percent  accuracy.  An  error  in  one  of 
these  lines  causes  a  "fatal  error,"  and  results  in  a  0  qualification  level,  or  a  "failed"  test. 

If  students  fail  the  300  level  test  three  times,  they  are  assigned  to  mandatory  night 
study,  allowed  one  more  practice  session,  and  then  assigned  up  to  three  more  tests  during 
the  next  session.  Regardless  of  the  outcome  of  these  tests,  the  next  three  typing 
assignments  serve  as  practice  for  the  400  function  level.  During  the  fourth  session, 
students  are  instructed  to  qualify  at  400  functions,  and  the  same  pattern  of  testing  ensues 
as  for  300  functions.  A  slight  change  in  procedure  occurs  if  a  student  fails  the  first  three 
tests  at  the  500  function  level.  Because  this  level  of  skill  is  required  to  progress  to  other 
parts  of  the  course,  a  student  must  remain  in  the  typing  session  until  he  achieves  the  500 
function  level.  This  can  take  more  than  2  weeks  of  day-long  typing,  with  mandatory  night 
study  after  three  test  failures.  If  progress  remains  inadequate,  the  student  ultimately 
goes  before  an  academic  review  board.  The  procedure  for  qualifying  at  600  functions  is 
the  same  as  for  300  and  400  functions;  and  that  for  qualifying  at  700  functions,  the  same 
as  for  500  functions.  However,  two  extra  practice  sessions  are  allowed  before  the  CMI 
computer  requires  a  student  to  qualify  at  the  final  level  of  700  functions.1 

Manual  Testing  Procedures.  A  number  of  steps  were  specific  to  the  manual  testing 
procedures.  Students  first  filled  out  administrative  forms--social  security  number,  name, 
course,  and  CMI  item  number.  The  computer  assigned  each  student  three  tests  that  were 
taken  at  one  of  five  testing  sessions  during  the  day.  Usually,  one  of  these  sessions  fell 
during  the  student's  2-hour  TTY  assignment.  For  the  first  test,  the  students  typed  for  5 
minutes  and  noted  the  number  of  functions  typed.  This  process  was  repeated  for  the 
second  and  third  tests,  after  which  students  counted  their  errors  on  ail  three  tests.  Next, 
the  students  waited  in  line  for  the  instructor  to  grade  the  tests  and  to  fill  out 
administrative  computer  forms  to  tell  the  CMI  system  whether  the  student  had  qualified. 
Students  then  waited  in  line  to  feed  the  administrative  forms  through  the  optical  mark 
reader,  and  waited  again  for  the  terminet  to  print  out  their  next  assignment. 

Automated  Testing  Procedures.  After  APT  had  been  implemented,  in  July  1978, 
testing  was  conducted  in  a  room  with  86  TTYs,  30  of  which  were  connected  to  a 
computer.  The  students  practiced  at  one  of  the  unconnected  machines  until  they  were 
ready  to  take  the  test.  They  then  obtained  a  hard  copy  of  the  CMI-assigned  test  and 
signed  on  at  one  of  the  computer  terminals  by  entering  the  same  information  required  on 
the  administrative  forms  under  the  manual  system.  The  test  began  when  the  first 
function  was  struck,  and  the  keyboard  locked  exactly  5  minutes  later.  A  local  computer 
scored  the  test  and  passed  the  results  on  to  the  CMI  system.  Test  results  and  the 
student's  next  assignment  were  displayed  at  the  student's  terminal.  The  immediate 
feedback  contained  such  information  as  the  number  of  errors,  the  lines  in  which  the  errors 
occurred,  the  number  of  functions  typed  before  the  sixth  error  or  before  a  fatal  error,  and 
the  level  of  qualification.  (Copy  for  an  entire  testing  session  is  shown  in  Figure  1.)  If  a 


‘Curriculum  revisions  implemented  after  the  R&D  covered  in  this  report  have  altered 
specific  qualifying  procedures,  although  the  same  final  qualification  standard  of  700 
functions  applies. 
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Figure  1.  Canplete  APT  testing  session 


student  failed  the  first  test,  the  CMI  system  immediately  assigned  up  to  two  more  tests. 
Tests  were  assigned,  timed,  scored,  and  recorded  quickly  and  accurately  without 
instructor  intervention. 

The  Error  Distribution  Report 

One  drawback  of  the  original  APT  system  was  that  it  had  few  provisions  for  helping 
students  identify  their  errors.  The  scoring  program  did  not  generate  feedback  on  what 
specific  errors  the  student  had  made.  Without  knowing  these  specific  errors,  students 
could  not  correct  them  as  efficiently  as  when  all  errors  were  identified.  To  correct  this 
deficiency,  NAVPERSRANDCEN  designed  a  computer-operated  error  distribution  report 
(EDR)  that  provided  detailed  feedback  on  student  typing  errors. 

Prior  to  development  of  the  EDR,  remedial  procedures  were  conducted  along 
traditional  lines.  During  remedial  night  study,  an  instructor  observed  a  student  typing 
certain  diagnostic  passages  for  30  minutes,  and  then  checked  the  typing  to  identify 
particular  problem  areas.  However,  he  had  only  that  one  sample  of  typing  on  which  to 
base  the  proper  remedial  assignments.  The  EDR  was  designed  to  automate  and  increase 
the  efficiency  of  this  function,  as  well  as  to  make  greater  use  of  the  computer  scoring 
program. 

EDRs  are  printed  on  one  page  for  each  student,  and  the  display  consists  of  two 
sections,  as  shown  in  Figure  2.  The  frequency  distribution  (top  section)  lists  the  21  most 
frequent  errors  by  the  intended  function  and  the  actual  function  struck.  In  the  example, 
we  see  that  the  student  made  26  errors  when  letter  (LTR)  was  the  intended  function 
indicated  by  the  keyboard  display.  He  omitted  the  character  (OC)  six  times,  hit  the  "D" 
and  the  space  bar  (SPC)  five  times,  hit  the  "E"  and  "N"  three  times,  and  hit  the  "M"  and 
"S"  two  times.  The  top  section  also  lists  the  error  patterns  on  the  last  test  taken. 

The  keyboard  diagram  (bottom  section)  shows  which  keys  or  groups  are  causing  the 
most  difficulty.  The  cumulative  history  provides  information  on  mistakes  the  student  has 
made  consistently  over  a  long  period  of  testing.  The  last  test-error  display  shows  whether 
problems  repeat  or  if  new  ones  are  appearing. 

Dvorak,  Merrick,  Dealey,  and  Ford  (1936)  have  said  that  a  mere  listing  of  errors  has 
little  or  no  meaning  for  the  student  if  he  does  not  understand  the  reason  for  these  errors. 
However,  Nathanson  (1929)  has  speculated  that  a  listing  of  persistent  keystroke  errors  is 
helpful  in  eliminating  these  mistakes.  While  the  EDRs  do  not  specify  the  underlying  cause 
of  the  errors  (e.g.,  poor  dexterity  or  lack  of  attention  to  copy),  they  do  indicate  what 
areas  are  causing  difficulties.  Directed  practice  should  then  help  the  student  overcome 
these  errors  regardless  of  the  cause.  Certainly  the  EDRs  can  help  the  instructors  discover 
where  typing  problems  occur.  They  can  then  observe  the  student  to  get  additional 
information  on  the  underlying  nature  of  a  particular  typing  problem. 

Because  of  programming,  distribution,  and  other  constraints,  an  EDR  containing 
updated  information  for  a  given  student  is  typically  available  no  earlier  than  the  morning 
following  the  day  of  a  test.  Since  instructional  material  is  interspersed  between  typing 
assignments,  a  student  could  not  practice  on  the  basis  of  immediate  feedback  anyway. 
Feedback  on  any  one  test  is  not  likely  to  be  of  great  help.  It  is  the  pattern  of  errors  after 
a  number  of  tests  have  been  taken  that  leads  to  a  dear  picture  of  where  the  difficulties 
lie.  Detailed  feedback  assures  that  students  can  concentrate  practice  on  problem  areas, 
overcome  problems  sooner,  and  complete  the  typing  portion  of  the  course  earlier. 


4 


UniJLAtlVE  tBRpo  patterns 


LCGfNC:  -  ,’r'>.L  K  £  T  FP  r  £  XCL  AV  A  T  I  0»I  POINT  C”  -  CAPPMGf.  Of  TURN  f  I G  :  MOUSES  K  C  Y  0**  -  CUE S T I  ON  “APr. 

LTo  :  l^tteps  xfY  i_£  -  l  I  ’  f  Ffn  spC  -  STArr  nr  :  o-itteo  CH'.Pxr  T  SR  1C  =  INSEPTfO  ci<a»actcp 

Figure  2.  Error  distribution  report. 


There  are  a  number  of  questions  related  to  the  implementation  of  APT.  How  often 
should  the  student  receive  feedback?  Are  there  more  efficient  ways  to  use  the 
information  obtained  from  the  automated  testing?  Could  more  efficient  TTY  practices 
and  remediation  techniques  decrease  training  time?  How  should  the  automated  testing 
and  feedback  be  integrated  into  the  central  CMI  system  functioning?  What  are  the  most 
effective  ways  the  school  can  use  detailed  feedback?  Even  if  APT  does  not  reduce  test¬ 
taking  time,  could  a  reduction  in  overall  training  time  be  great  enough  'o  justify  longer 
test  taking  time?  These  questions  apply  not  only  to  TTY  operations  in  the  RM  school,  but 
also  to  training  on  other  keyboard  devices  for  other  ratings  with  similar  training 
requirements  (e.g.,  Communications  Technicians,  Data  Specialists,  Yeomen,  and  Person- 
nelmen). 

Previous  research  in  the  area  of  APT  is  somewhat  limited.  Although  many  studies 
have  been  conducted  to  determine  the  most  efficient  methods  of  teaching  typing  (Dierks, 
1977;  Dixon,  1976;  Krag  &  Van  Brunt,  1970;  Pask,  1960;  Peterson  &  Staples,  1969;  Rainey, 
1977;  Sharkey  <Sc  Thomas,  1973;  Sherrill,  1976;  Showel,  1974;  Wolcott,  1976),  none  could  be 
found  that  dealt  directly  with  presenting  student  error  histories.  The  studies  focused  on 
the  use  of  audiovisual  presentation,  group  and  individual  pacing  devices,  and  instructional 
techniques.  The  practice  material  was  geared  to  the  number  of  errors,  not  to  the  specific 
types  of  errors  made. 

Objectives 

The  objectives  of  this  effort  were: 

1.  To  determine  whether  test-related  activities  take  less  time  under  APT  than 
under  manual  testing  conditions. 

2.  To  determine  whether  training  time  can  be  reduced  by  different  applications  of 
EDR  use,  and,  if  so,  the  most  feasible  and  effective  techniques  for  distributing  EDRs  to 
students  and  instructors. 


METHOD 

To  determine  the  effects  of  APT  and  EDRs  on  training,  two  experiments  were 
conducted.  These  experiments  are  described  below. 

Experiment  I 

In  the  first  experiment,  the  times  required  for  various  test-related  activities  under 
the  manual  and  automated  systems  were  compared  to  determine  the  absolute  difference 
in  testing  times  between  manual  and  automated  testing  of  TTY  skills  within  a  CMI 
environment. 

Subjects 

The  subjects  were  students  enrolled  in  the  RM  "A"  school  during  10  sessions  in  April 
1978,  when  manual  testing  procedures  were  used,  and  during  10  sessions  in  August  1978, 
when  automated  testing  procedures  were  used.  Manual  testing  involved  4  to  12  subjects 
per  session,  depending  on  the  number  of  students  ready  to  take  a  test  during  a  given 
session.  Automated  testing  involved  only  one  student  per  session,  since  such  testing  is 
individualized. 


Procedure 


During  the  experiment,  records  were  made  of  the  time  required  to  complete  test- 
related  activities,  including  (1)  filling  out  administrative  forms,  (2)  announcing  the  test 
(instructor),  (3)  actual  testing,  (4)  self-grading,  (5)  instructor  grading,  (6)  completing 
administrative  forms  (instructor),  (7)  feeding  the  forms  through  the  optical  mark  reader 
(students),  and  (S)  obtaining  the  next  assignment.  The  timing  of  similar  events  was 
recorded  for  students  in  the  APT  group. 

Experiment  II 

In  the  second  experiment,  groups  of  students  were  given  EDRs  under  different 
conditions  to  determine  (1)  whether  EDRs  could  reduce  the  time  they  needed  to  reach  the 
criterion  level  in  TTY  training,  and  (2)  if  so,  the  most  efficient  techniques  for  employing 
EDR  feedback.  It  was  predicted  that  the  more  frequent  the  updated  reports,  the  more 
efficient  remediation  and  practice  would  be,  and  the  shorter  the  TTY  training  time. 

Although  the  main  dependent  measures  in  this  experiment  were  performance  related, 
another  measure  studied  was  the  student's  attitude  toward  the  CMI  system,  APT,  and  the 
EDRs.  It  was  hoped  that  students  would  react  favorably  to  fair  and  accurate  testing  and 
would  appreciate  receiving  current  and  accurate  information  about  their  typing  errors. 

Subjects 

The  subjects  for  this  experiment  were  60  students  in  the  RM  "A"  School.  Students 
were  randomly  assigned  to  one  of  three  groups  as  they  first  entered  the  testing  room: 

1.  Students  in  the  Daily  (D)  Group  were  to  receive  EDR  feedback  on  the  day  after  a 
test  was  given. 

2.  Students  in  the  Remedial  (R)  Group  were  to  receive  EDR  feedback  only  during 
their  second  remedial  session. 

3.  Students  in  the  Never  (N)  Group  were  to  receive  no  feedback  via  EDRs. 

Materials  and  Apparatus 

The  experiment  took  place  in  a  large  room  with  86  TT-47  teletypewriters.  An  optical 
scanner  (OpScan  17)  for  manual  input  to  the  CMI  system  was  in  the  center  of  the  room 
beside  a  General  Electric  Terminet  1200  that  printed  out  student  assignments  from  the 
main  CMI  computer.  The  30  TTY  machines  in  the  front  of  the  room  were  connected  on¬ 
line  to  a  Honeywell  computer  for  scoring  purposes. 

Procedure 


Due  to  the  constraints  of  the  computer  system,  the  EDRs  for  tests  taken  each  day 
could  not  be  processed  until  after  midnight,  and  thus  were  not  available  until  mid-morning 
of  the  following  day.  For  students  in  Group  D,  EDRs  were  generated  automatically  every 
day.  If  the  student  had  taken  a  test  the  previous  day,  he  was  given  the  EDR  the  next  time 
he  reported  for  typing.  For  those  in  Group  R,  EDRs  were  distributed  during  the  second 
consecutive  appearance  for  remedial  study,  since  they  could  not  be  processed  until  after 
the  first  reporting.  Thus,  if  students  never  reported  to  remedial  study,  or  did  so  only 
once,  they  would  not  receive  an  EDR.  Students  in  Group  N  received  no  EDRs. 
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All  participating  students  were  told  that  an  experiment  was  being  conducted  to 
evaluate  the  computer  system.  The  experimenter  described  the  reports  and  their  use  to 
students  the  first  time  they  were  distributed,  but  from  then  on  the  students  studied  the 
EDRs  by  themselves.  The  reports  were  distributed  until  all  participating  students  had 
qualified  at  the  700  function  level. 

Attrition  data,  plus  dependent  measures  related  to  TTY  performance,  were  available 
from  CMI  response-history  records.  The  number  of  TTY  sessions,  the  number  of  TTY 
tests  taken,  and  the  number  of  hours  spent  in  the  TTY  sessions  before  achieving  criterion 
levels  of  performance  were  recorded  when  the  first  test  was  taken  (Tl),  and  at  each  of 
the  five  qualification  levels  (300  to  700  functions).  Additionally,  in  the  third  week  after 
data  collection  began,  questionnaires  regarding  attitudes  toward  CMI  and  APT  were 
distributed  to  students  in  the  typing  room  regardless  of  whether  or  not  they  had 
participated  in  the  experiment.  Ten  students  who  had  received  EDRs  were  given  a 
questionnaire  that  included  questions  on  the  EDRs.  Emphasis  was  placed  on  the  fact  that 
all  questionnaires  were  anonymous.  Students  filled  out  the  questionnaire  on  their  own 
time.  The  questions  included  in  the  questionnaires  are  listed  in  Figure  3. 


General; 

1. 

2. 

CMI: 

1. 

2. 


3. 

4. 

APT: 

1. 

explain. 

2. 

3. 

4. 

EDR: 

1. 

2. 

3. 


How  long  have  you  been  in  school? 

At  what  function  level  are  you  now  qualified? 

What  do  you  like  about  the  CMI  teaching  method? 

What  do  you  dislike  about  the  CMI  teaching  method? 

How  does  the  CMI  method  compare  with  having  human  teachers? 

In  what  ways  could  the  CMI  course  be  improved? 

Does  the  computer  scoring  of  TYPING  tests  seem  fair  and  accurate?  Please 

Do  you  get  your  next  assignment  quickly  after  a  TYPING  test? 

Do  you  understand  how  the  computer  scores  TYPING  tests?  Please  explain. 

How  could  the  way  the  computer  scores  TYPING  tests  be  improved? 

Do  the  typing  error  analysis  reports  made  by  the  computer  seem  helpful? 

Do  you  understand  how  to  use  the  typing  error  analysis  reports? 

How  could  the  typing  error  analysis  reports  be  improved  or  made  better? 

Figure  3.  Questions  included  in  CM1/APT  attitude  survey. 
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RESULTS 


Experiment  1 

Table  1  provides  the  mean  times  required  for  test-related  activities  under  manual 
testing  conditions- -which  always  involves  three  tests--and  under  automated  testing 
conditions — which  may  involve  one,  two,  or  three  tests.  As  shown,  students  required  2 
minutes  to  enter  course  information  manually  on  an  administrative  form.  They  also 
required  2  minutes  to  enter  course  information  on  the  TTY  terminal--but  only  when  just 
one  test  was  involved.  If  a  student  needed  to  take  a  second  or  third  test,  the  time 
required  increased  to  4  and  6  minutes  respectively  (the  mean  number  of  tests  taken  by 
students  under  APT  was  2.8). 


Table  1 

Time  Required  (Minutes)  for  Test-related  Activities 
Under  Manual  and  Automated  TTY  Test  Conditions 


TTY  Test  Condition3 

Activity 

Manual 
(3  tests) 

(l  test) 

Automated 
(2  tests) 

(3  tests) 

Completing  Course  Information 

2 

2 

4 

6 

Taking  Tests 

Waiting  for  Scores: 15 

20 

5 

10 

15 

1  student 

1 

1 

2 

3 

(15  students) 

(8) 

(1) 

(2) 

(3) 

(30  students) 

(15) 

(1) 

(2) 

(3) 

Waiting  for  Next  Assignment0 

3 

0 

0 

0 

Total—  1  student 

26 

8 

16 

24 

15  students 

33 

8 

16 

24 

30  students 

40 

8 

16 

24 

aUnder  the  manual  testing  condition,  three  tests  were  always  given.  Under  the  automated 
condition,  if  the  student  passed  the  first  test,  no  more  tests  were  given.  The  mean 
number  of  tests  taken  by  students  under  APT  was  2.8. 

^Under  the  manual  condition,  students  had  to  wait  in  line  for  their  score  about  one  half 
minute  each;  thus,  the  larger  the  number  of  students,  the  longer  the  time  required  to 
wait  in  line.  Under  APT,  students  received  their  scores  automatically;  thus,  it  made  no 
difference  how  many  students  were  involved. 

cUnder  APT,  students  received  their  next  assignment  at  the  seme  time  they  received 
their  grades.  Thus,  no  extra  time  Was  required  for  this  activity. 

There  was  a  significant  difference  in  the  amount  of  time  required  for  manual  and 
automated  testing.  As  shown,  students  in  the  manual  group  required  20  minutes,  which 
includes  the  time  spent  getting  test  booklets,  marking  the  total  functions  between  tests, 
and  taking  the  tests  themselves.  In  the  APT  group,  a  test  took  exactly  5  minutes. 


If  the  student  passed  the  first  test,  this  was  the  total  testing  time.  However,  if  he  needed 
to  take  a  second  or  third  test,  the  times  increased  to  10  and  15  minutes  respectively, 
which  are  still  10  and  5  minutes  less  than  the  time  required  for  manual  testing. 

The  next  activity,  waiting  for  scores,  yields  the  largest  difference  between  groups. 
Actual  scoring  for  the  APT  group  was  virtually  instantaneous,  no  matter  how  many 
students  were  involved,  but  it  took  1  minute  to  receive  the  printout  and  retrieve  the  next 
assignment.  For  the  manual  test  group,  the  time  varied  with  the  number  of  students  being 
tested.  Generally,  as  the  number  of  students  increased,  the  average  time  required  to 
complete  scoring  a  student's  tests  increased.  The  actual  scoring  time  was  about  .5  minute 
per  student  when  10  or  more  students  were  tested  at  one  session. 

Finally,  no  extra  time  was  required  to  provide  the  next  assignment  to  APT  students. 
By  the  time  the  student  TTY  terminal  finished  printing  the  test  results,  the  CMI  Learning 
Center  terminet  had  printed  the  next  assignment.  The  manual  test  students,  however,  had 
to  wait  for  the  instructor  to  code  their  administrative  forms,  for  a  chance  to  feed  their 
forms  to  the  optical  scanner,  and  then  to  receive  the  next  assignment. 

The  minimum  time  for  manual  test  students  to  complete  all  activities  (assuming  I 
student)  was  26  minutes;  for  APT  students,  the  minimum  time  (assuming  1  test)  was  8 
minutes.  Even  when  all  three  tests  were  taken  by  APT  students,  the  mean  time  required 
was  24  minutes.  This  is  still  less  than  the  26  minutes  a  manual  test  student  requires  to 
complete  the  testing  cycle  (even  if  scoring  were  immediate),  and  much  less  than  the  40 
minutes  required  when  30  tests  must  be  scored. 

Experiment  II 

Similarity  of  Subject  Groups 

As  a  check  on  group  equality,  group  mean  Armed  Services  Vocational  Aptitude 
Battery  (ASVAB)  qualifying  scores  were  compared,  and  no  significant  differences  were 
found.  Prior  typing  experience  could  also  influence  TTY  progress,  since  students  with 
typing  skills  usually  do  not  need  the  full  eight  practice  sessions  to  pass  the  first  test  for 
300  functions.  Thus,  to  check  on  the  equality  of  groups  with  respect  to  typing  experience, 
the  mean  numbers  of  TTY  sessions  up  to  and  including  the  first  test  were  compared.  The 
mean  numbers  of  hours  spent  in  basic  typing  instruction  prior  to  the  first  test  were  16.8, 
15.3,  and  17.7  for  D,  R,  and  N  groups,  respectively.  These  differences  were  not 
significant,  again  indicating  similarity  of  the  subject  groups. 

Attrition 


Fifteen  of  the  60  subjects  were  dropped,  for  an  overall  attrition  rate  of  25  percent. 
This  included  three  (15%)  from  Group  D,  five  (25%)  from  Group  R,  and  seven  (35%)  from 
Group  N  (see  Table  2).  Although  some  students  were  dropped  for  academic  reasons ,  and 
others,  for  motivational  or  disciplinary  reasons,  persons  in  both  attrition  categories  were 
having  difficulty  in  the  typing  portion  of  the  course.  In  fact,  according  to  the  Service 
School  Command  in  San  Diego,  12  of  the  15  attrites  (80%)  were  having  problems  with 
typing  when  they  were  dropped.2  This  included  two  (10%)  from  Group  D,  and  five  (25%) 
from  Groups  R  and  N.  Attrites  spent  an  average  of  58.6  days  in  the  course. 


2Chief  of  Naval  Technical  Training  RM  "A"  Attrition  Message  052220Z  December 
1978. 
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Table  2 


Performance  Data  for  Students  in  EDR  Groups 


TTY  Qualification  Level 


Group 

Tla 

300 

400 

500 

600 

700 

Attrition 

D  (Daily) 

20 

20 

20 

20 

20 

17 

R  (Remedial) 

19 

19 

19 

18 

17 

15 

N  (Never) 

19 

19 

19 

18 

14 

13 

Mean  Number  of  Hours  to  TTY  Criterion** 


Actual  Ns: 

D 

17 

24 

32 

43 

67 

80 

R 

15 

25 

34 

51 

66 

74 

N 

18 

27 

44 

57 

68 

84 

Adjusted  Ns:c 

D 

17 

24 

32 

38 

49 

64 

R 

15 

25 

34 

51 

56 

65 

N 

18 

28 

44 

57 

68 

84* 

Mean  Number  of  TTY  Sessions  to  Criterion^ 


D 

7.5 

9.9 

12.6 

15.6 

23.2 

25.7 

(2.6) 

(4.0) 

(5.2) 

(4.9) 

(8.0) 

(7.4) 

R 

7.4 

11.6 

14.0 

16.0 

23.1 

24.4 

(2.6) 

(5.2) 

(5.4) 

(4.7) 

(9.6) 

(7.0) 

N 

8.2 

11.7 

16.1 

18.1 

22.5 

26.0 

(2.0) 

(3.8) 

(5.0) 

(3.6) 

(7.1) 

(6.0) 

Mean  Number  of  TTY  Tests  to  Criterion*^ 


D 

5.0 

9.1 

15.8 

25.2 

32.0 

— 

(3.1) 

(7.0) 

(12.6) 

(17.0) 

(21.8) 

R 

9.2 

14.9 

24.2 

33.1 

39.6 

— 

(7.1) 

(11.6) 

(16.1) 

(21.1) 

(24.8) 

N 

7.8 

15.9 

26.8 

29.7 

40.4 

— 

(6.6) 

(13.8) 

(18.0) 

(20.9) 

(22.7) 

aPoint  at  which  first  test  was  taken. 

**Hours  were  rounded  to  the  nearest  whole  hour. 

cAt  the  500,  600,  and  700  levels,  the  numbers  of  students  in  each  group  were  equalized  to 
test  for  contamination  of  data. 

^Nos.  in  parentheses  are  standard  deviations. 
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Number  of  Hours  in  TTY  Sessions  to  Criteria 


Table  2,  which  includes  the  mean  numbers  of  hours  students  spent  in  TTY  sessions 
before  reaching  the  various  qualification  levels,  shows  that  the  difference  between  Groups 
D  and  N  increased  up  to  the  500  level.  However,  at  the  600-function  level,  data  from  all 
three  groups  converged,  primarily  because  of  the  increase  in  time  required  for  Groups  D 
and  R  to  reach  that  level,  rather  than  from  a  change  in  the  rate  of  Group  N.  While  the 
groups  separate  again  at  the  700  level,  the  differences  are  not  large. 

The  cause  for  the  convergence  at  the  600  level  may  be  related  to  the  differential 
attrition  rates  among  the  groups,  which  may  have  contaminated  the  data.  Since  most  of 
the  attrites  were  dropped  for  taking  an  excessive  amount  of  time  to  reach  the  500,  600,  or 
700  levels,  it  may  be  that  the  groups  with  lower  attrition  rates--D  and  R--had  a  higher 
proportion  of  poor  students  at  those  levels.  The  EDRs  may  have  helped  the  marginal 
students  in  Group  D,  and,  to  a  lesser  extent,  those  in  Group  R,  at  least  enough  to  finish 
the  course.  If  those  marginal  students  had  been  dropped,  the  mean  time  for  Groups  D  and 
R  may  have  been  substantially  lower. 

To  investigate  this  possibility,  the  numbers  of  students  in  each  group  at  the  500,  600, 
and  700  function  level  were  equalized.  Since  Group  N  had  the  lowest  number,  the  other 
groups  were  reduced  to  that  number.  In  this  process,  data  for  the  slowest  students  from 
Group  D  and  R  were  deleted  since  it  was  assumed  that  these  students  would  be  most 
similar  to  the  students  who  dropped  out  from  Group  N.  The  resulting  data  are  presented 
in  Table  2,  which  shows  that  it  took  the  students  in  Group  N  a  significantly  longer  time  to 
reach  the  700  level  than  it  did  those  in  Groups  D  and  R  (t  (37)  z  1.94,  p  <  .05). 

Number  of  TTY  Sessions  to  Criteria 

Results  of  an  analysis  of  variance  on  the  mean  numbers  of  TTY  sessions  required  bv 
group  members  to  reach  criterion  revealed  no  significant  differences  between  groups, 
when  using  either  actual  or  adjusted  data.  The  TTY  session  data  shown  in  Table  2, 
however,  may  display  a  ceiling  effect.  At  the  RW  school,  students  are  allowed  only  21 
sessions  to  reach  the  500  level;  and  33  sessions,  to  reach  the  700  level.  Records  showed 
that  one  third  of  the  students  required  21  sessions  to  reach  the  500  level.  Of  those,  all 
but  two  either  required  33  sessions  to  reach  the  700  level  or  failed  to  reach  it  and  were 
dropped  from  the  course.  Therefore,  without  the  limit  on  the  number  of  sessions  at  the 
500  and  700  levels,  some  students  would  have  spent  many  more  sessions  trying  to  qualify 
at  these  levels. 

Number  of  TTY  Tests  to  Criteria 

Data  on  the  mean  numbers  of  TTY  tests  taken  (Table  2)  indicate  the  same  trend  in 
results  as  the  previous  measure;  these  were  no  significant  differences  among  the  means. 
Again,  this  picture  was  not  changed  by  adjusting  the  data  for  attrition.  This  might  have 
been  expected,  since  there  were  large  differences  in  individual  testing  strategies.  Some 
students  wanted  to  take  tests  often,  while  others  tested  only  when  required. 

Attitude  Survey 

Seventy-six  attitude  surveys  were  returned,  including  eight  of  the  ten  with  questions 
on  EDR.  Table  3,  which  provides  responses  by  student  qualification  level,  shows  that, 
although  students  were  generally  positive  toward  both  CMI  and  APT,  they  were  more 
positive  toward  the  APT  portion  of  CMI  than  to  the  system  in  general.  Attitudes  seemed 
to  be  strongly  affected  by  what  qualification  level  the  respondents  were  trying  to  achieve 


12 


Table  3 


Responses  to  CMI/APT  Attitude  Surveys 


Respondents  at  TTY  Qualification  Level 

Survey 

300 

400 

500 

600 

700 

Topic 

(%) 

(%) 

(96) 

(96) 

(96) 

CMI  (N  =  76): 

Positive 

57 

68 

43 

51 

37 

Negative 

APT  (N  =  76): 

43 

32 

57 

49 

63 

Positive 

57 

73 

54 

74 

41 

Negative 

EDR  (N  =  8): 

43 

27 

46 

26 

59 

Positive 

100 

100 

100 

100 

100 

Negative 

0 

0 

0 

0 

0 

at  the  time  of  responding.  Attitudes  did  decline  at  the  500  and  700  levels.  When  students 
were  attempting  to  reach  700  level,  their  attitude  was  clearly  negative.  All  eight 
students  who  returned  surveys  with  EDR  questions  indicated  that  they  were  satisfied  with 
the  reports. 


DISCUSSION  AND  CONCLUSIONS 

The  experiments  show  that  APT  with  proper  feedback  aids  in  teaching  TTY  skills. 
The  APT  testing  time  was  shorter  than  manual  testing  time,  as  shown  by  the  first 
experiment,  and  frequent  distribution  of  the  EDRs  was  beneficial.  Students  receiving 
feedback  not  only  finished  the  typing  portion  of  the  course  faster  than  other  students,  but 
attrition  from  typing  problems  was  lower. 

One  reason  why  APT  speeded  testing  was  that  students  under  APT  did  not  need  to 
take  all  three  tests  if  they  passed  one  of  the  first  two.  Also,  they  had  the  opportunity  to 
practice  between  individual  tests  if  their  performance  on  the  first  or  second  test 
indicated  extra  practice  was  needed.  Students  could  take  tests  when  they  felt  they  were 
ready,  not  just  whenever  the  test  was  given,  while  the  manual  group  was  arbitrarily 
required  to  take  all  three  tests.  An  additional  advantage  for  the  APT  students  was  the 
option  to  terminate  a  test  if  they  realized  they  had  already  made  too  many  errors  before 
the  5-minute  period  was  over.  APT  gave  students  more  time  to  practice  and  improve 
their  typing  skills.  It  also  allowed  the  instructor  to  spend  more  time  helping  students  with 
typing  problems  instead  of  performing  administrative  scoring  duties. 

The  morale  factor  is  also  important.  APT  eliminates  the  irritation  students  feel  at 
delays  for  test  grading,  and  the  tedious  responsibility  instructors  face  in  grading  tests. 
Because  instructors  are  free  of  irksome  tasks,  they  may  feel  better  about  their  job  and 
can  teach  more  effectively. 
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Probably  the  most  important  by-product  of  APT  is  the  ability  to  provide  detailed 
feedback  concerning  typing  errors  to  both  students  and  instructors.  Although  EDRs  were 
not  available  until  the  day  after  a  test  was  taken,  students  were  usually  given  nontyping 
assignments  after  TTY  tests  and  immediate  feedback  could  not  be  used  anyway.  EDRs 
were  usually  available  by  the  time  the  student  got  his  next  TTY  assignment.  Most 
students  receiving  EDRs  took  advantage  of  the  feedback  by  studying  them  carefully  and 
concentrating  practice  on  the  areas  of  greatest  error.  This  concentration  on  specific 
areas  would  seem  to  be  the  reason  for  the  improved  performance  of  the  EDR  groups  over 
groups  not  receiving  EDRs.  The  performance  differences  in  the  time  required  to  reach 
the  700-function  level  were  significant  and  nearly  all  the  other  comparisons  were 
consistent.  The  evidence  is  less  clear  when  the  number  of  sessions  is  considered,  but  this 
inconsistency  seems  to  be  a  ceiling  effect  due  to  the  restriction  in  the  number  of  sessions 
students  were  permitted. 

General  approval  of  the  APT  and  EDR  programs  by  the  students  was  an  added 
benefit.  The  survey  data  showed  that  more  students  responded  favorably  to  APT  than  to 
the  CMI  system  itself,  indicating  that  they  considered  APT  improved  the  course.  This 
attitude  was  pronounced,  although  it  was  clear  that  some  students  did  not  understand 
computer  operation,  challenged  the  accuracy  of  computerized  scoring,  and  even,  in  some 
cases,  insisted  it  was  inaccurate.  These  misunderstandings  indicate  that  the  test  scoring 
process  must  be  explained  and  that,  if  students  were  better  informed  on  computer 
operations,  their  attitudes  would  improve  and  the  acquisition  of  knowledge  and  skills 
would  be  faster  and  better. 

The  survey  responses  showed  a  favorable  reaction  to  the  EDRs.  All  students 
completing  the  survey  said  the  reports  were  worthwhile,  and  a  large  number  of  students 
not  in  the  experiment  requested  reports  on  a  regular  basis.  Many  students  were  eager  to 
see  their  next  report  and  studied  them  carefully.  They  were  pleased  to  receive  useful 
information  from  the  computer.  It  was  clear  that  the  feedback  lessened  the  boredom  of 
typing  practice,  increased  motivation,  and  heightened  interest  in  acquiring  skill. 

It  is  possible  that,  without  EDRs,  some  students  with  severe  typing  problems  might 
have  become  frustrated  with  the  course,  developed  severe  TTY  problems,  and  dropped  out 
of  the  course.  Conversely,  when  the  EDRs  were  available,  they  may  have  given  some 
students  needed  encouragement  through  the  feedback  that  showed  their  progress  in 
overcoming  errors.  The  attrition  data  supports  this  assumption — the  attrition  rates  for 
Groups  D,  R,  and  N  were  15,  25,  and  35  percent  respectively.  The  D  group  had  no 
dropouts  for  motivational  or  disciplinary  reasons. 

Use  of  the  EDRs  to  guide  practice  may  have  helped  marginal  students  to  stay  in  the 
course  until  completion.  The  benefit  of  saving  these  students  has  an  associated  cost;  they 
need  more  time  to  complete  the  course.  In  fact,  the  performance  times  of  the  marginal 
students  who  would  have  dropped  out  without  the  EDRs  probably  accounted  for  the 
relatively  high  mean  performance  times  for  the  EDR  groups  beyond  the  400-function 
level.  The  marginal  students  undoubtedly  accounted  for  the  20  percent  differential 
attrition  between  groups  D  and  N.  After  adjusting  the  data  for  differential  attrition,  it  is 
clear  that  the  EDR  groups  were  superior  to  the  N  group,  taking  much  less  time  to  reach 
the  700-function  level. 

To  understand  the  importance  of  the  time  saved,  the  associated  cost  avoidance  must 
be  examined.  The  savings  in  time  to  reach  the  highest  required  level  of  performance 
(700-functions)  between  groups  D  and  N  was  19.8  hours,  or  just  under  three  7-hour  CMI 
training  days,  based  on  the  data  adjusted  for  differential  attrition.  With  the  marginal 
students  included,  however,  the  difference  was  minimal.  Since  the  large  difference  was 


computed  with  only  13  of  the  20  students,  the  actual  savings  apply  only  to  those  65 
percent  of  the  students  who  were  not  marginal.  The  cost  per  student  enrolled  per  day  in 
the  RM  "A"  school,  according  to  the  specialized  training  costs  group  under  the  Chief  of 
Naval  Education  and  Training,  was  projected  to  be  $83.92  in  1979,  including  military  and 
civilian  pay,  machine  maintenance,  and  general  overhead.  Data  in  this  present  report  are 
based  on  a  small  experimental  sample  of  students.  However,  if  the  results  were  extended 
over  the  projected  1979  enrollment  of  3106  students  (RM  school  figure),  the  cost  saving 
would  amount  to  $479,000  ($83.92  per  student  day  x  2.83  days  x  3106  students  x  .65).  The 
potential  for  additional  Navy-wide  savings  could  be  exploited  if  APT  techniques  were 
developed  and  implemented  in  other  schools  that  have  similar  typing  requirements. 

Based  on  the  preliminary  results  of  this  effort,  procedures  for  the  optimal  use  of 
EDRs  were  developed  through  the  coordinated  efforts  of  the  RM  school,  IPDC,  and 
N  AVPERSRANDCEN.  These  procedures  included  guidelines  for  distribution  of  the  reports 
and  use  during  classroom  remediation.  Specifically,  under  these  procedures,  instructors 
receive  reports  on  students  who  fail  to  qualify  at  a  required  level  of  performance.  The 
students  are  also  given  a  copy  of  the  EDR  and  suggestions  on  what  to  practice.  According 
to  these  guidelines,  students  are  given  the  report  and  counseling  even  if  they  never  report 
to  night  study.  This  is  almost  equivalent  to  providing  the  student  with  "daily"  reports, 
since  "daily"  operationally  refers  to  any  time  a  test  is  taken.  The  student  gets  little  help 
from  feedback  on  a  test  that  is  "passed,"  since  so  few  errors  are  made.  Therefore,  giving 
students  reports  after  "failed"  tests  provides  them  with  nearly  as  much  information  as 
giving  reports  after  every  test. 

At  the  completion  of  these  experiments,  the  RM  school  was  following  these 
guidelines  and  the  instructors  were  satisfied  with  them.  The  programs  developed  in  this 
effort  represent  a  significant  advancement  in  instructional  technology  used  in  automated 
performance  testing.  The  APT  technology  has  demonstrated  the  potential  to  effect 
substantial  savings  in  the  RM  school  through  reduced  training  time  and  less  attrition,  and 
could  now  be  applied  to  performance  skills  involved  in  other  CMI  courses. 


RECOMMENDATIONS 

1.  The  RM  school  should  continue  to  use  APT  and  EDRs. 

2.  The  Chief  of  Naval  Technical  Training  and  the  Service  School  Commands  should 
exploit  the  potential  cost-effectiveness  of  APT  at  other  Navy  technical  training  schools. 
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